[feature request] listen on 127.0.0.1 only by default

[cphlipot]

Currently llama-swap listens on all network interfaces, meaning anyone on the same network as the hosting PC can connect to the llama-swap service. this is potentially problematic for some users because:

• it is never disclosed to users it is being served outside their local PC, potentially creating unintended security/privacy issues.
• outside users can submit any POST/GET request including the ability to read logs, or existing chat/completions activity
• By default llama-swap has no authentication.

I recommend we switch to listen on 127.0.0.1:8080 as default behavior to keep things private by default, and instead provide users instructions on how to open access more widely if they desire.

[mostlygeek]

This is a one line change so it's trivial to implement. I don't think this is the right change to make because:

• changing defaults will break someone's set up unexpectedly. Breaking backwards compatibility intentionally is something that llama-swap has rarely done. The goal is to never break people's configurations.
• llama-swap is meant to be a server for many clients. Many users are running it on a GPU server on their network serving multiple clients.
• it's easy enough to specify, -listen http://localhost:8080 to limit access.

[cphlipot]

I would think this may need to be more than a 1 line change as we'd likely also want to update the docker image to work with examples, since then intention there is to usually serve the port outside the container.

other projects i have seen have made the decision to change the default when this issue was raised by users. eg. unslothai/unsloth#4684 (comment)

If we can't change this could we do anything at least to make it clear to users that the default behavior doesn't keep your models and conversations private? I think that is a large part of the issue here, that current behavior was unexpected.

[mostlygeek]

An important distinction is that llama-swap is meant to be a server first. If llama-swap-desktop ever existed it would default to listening on localhost or a unix domain socket.

[knguyen298]

I feel like the onus of ACL for a server-oriented application should fall on the user, not on the developer.

If you're using Docker, why are you not using the following:

ports:
  - 127.0.0.1:8080:8080

If you have it installed bare metal, why are you not using an OS-level firewall like ufw or Windows Firewall?

Also regarding this statement:

it is never disclosed to users it is being served outside their local PC, potentially creating unintended security/privacy issues.

is inaccurate. If you understand what you're setting up, e.g. using Docker with -p 8080:8080, it is evident that this will open port 8080 to anything that can reach your device.

[cphlipot]

I feel like the onus of ACL for a server-oriented application should fall on the user, not on the developer.

I don't necessarily disagree here, but IMHO it would be better to not serve on external interfaces by default in this case. Many of the other projects llama-swap is intended to wrap(eg llama-server, sd-server, etc.) follow this and bind only to 127.0.0.1 by default.

@knguyen298 My concern mostly about preventing less knowledgeable or inexperienced users from accidentally over exposing information. I think you ask several questions about why doesn't the user do X, but in many of those cases they should, but don't because it is not default and they don't know enough to know they should do them.

Changing defaults is one way to help combat this, although it sounds like there are concerns around backwards compatibility, so possible that may not be an option. If not i think it may still be beneficial to find a better way to manage user expectations here.

If you understand what you're setting up, e.g. using Docker with -p 8080:8080, it is evident that this will open port 8080 to anything that can reach your device.

Also just wanted to clarify the original intent of this request was regarding the default of the CLI itself, not Docker. As you pointed out docker already requires an explicit option to be thrown "-p 8080:8080" in order to expose this. It is not immediately clear that running the llama-swap command with no arguments will also do the same.

[mostlygeek]

A lot of good points. A good middle ground would be for llama-swap to maintain the current default and output a notice when listening on more than 127.0.0.1/localhost.